Travel controller and method for travel control

ABSTRACT

A travel controller detects objects from an environmental image representing surroundings of a vehicle capable of traveling under autonomous driving control satisfying a predetermined safety standard; detects a looking direction of a driver of the vehicle from a face image of the driver; identifies an object in the looking direction of the driver out of the objects; stores the identified object and a situation condition indicating the situation at detection of the identified object in a memory in association with each other when a danger avoidance action performed by the driver is detected during travel of the vehicle; and changes the predetermined safety standard so that the driver can feel safer, in the case where an object stored in the memory is detected during travel of the vehicle under the autonomous driving control and where the situation at detection of the object satisfies the situation condition.

FIELD

The present disclosure relates to a controller and a method forautomatically controlling travel of a vehicle.

BACKGROUND

A travel controller that automatically controls travel of a vehicle onthe basis of environmental images generated by a camera mounted on thevehicle is known. The travel controller detects an object in an areaaround the vehicle from an environmental image, and controls travel ofthe vehicle so as to avoid collision with the object.

A data processing system described in Japanese Unexamined PatentPublication No. 2018-101400 (hereafter “Patent Literature 1”) determinesa user's driving behavior and preference, based on driving statisticscollected while an autonomous vehicle is driven in manual driving mode,and generates the user's driving profile, based on the user's drivingbehavior and preference. The driving profile generated by the techniqueof Patent Literature 1 in a driving scenario, such as route selectionand a lane change, is used for travel control of the autonomous vehiclein a similar driving scenario.

SUMMARY

Even if a tendency for a driver to perform predetermined drivingbehavior in a certain driving scenario is recognized, it is notnecessarily clear whether the driver paid attention to the drivingscenario to perform the driving behavior. For example, in the case wheredriving slowly at an intersection without traffic signals is recognizedas driving behavior, the driver may drive slowly, paying attention to apedestrian near an intersection rather than the intersection itself. Insuch a case, driving without slowing down, for example, when passing aplace other than an intersection, where a pedestrian is about to crossthe road may make the driver feel uneasy, and thus is not necessarilyappropriate travel control.

It is an object of the present disclosure to provide a travel controllerthat can control travel of a vehicle so that its driver does not feeluneasy.

A travel controller according to the present disclosure includes aprocessor configured to detect one or more objects from an environmentalimage representing surroundings of a vehicle capable of traveling underautonomous driving control satisfying a predetermined safety standard;detect a looking direction of a driver of the vehicle from a face imagerepresenting a face region of the driver; and identify an object in thelooking direction of the driver out of the one or more objects. Theprocessor of the travel controller is further configured to store theidentified object and a situation condition indicating the situation atdetection of the identified object in a memory in association with eachother when a danger avoidance action performed by the driver to avoiddanger is detected during travel of the vehicle; and change thepredetermined safety standard so that the driver can feel safer, in thecase where an object stored in the memory is detected during travel ofthe vehicle under the autonomous driving control and where the situationat detection of the object satisfies the situation condition.

In the travel controller according to the present disclosure, thesituation condition preferably includes at least the distance from thevehicle to the identified object.

The processor of the travel controller according to the presentdisclosure is preferably further configured to make a learned situationcondition, based on a plurality of situation conditions stored atdifferent times in association with an object stored in the memory. Thelearned situation condition indicates a situation in which the dangeravoidance action is detected at detection of the object. The processorof the travel controller, at changing the safety standard, preferablydetermines that the situation at detection of the object satisfies thesituation condition, when the situation at detection of the objectsatisfies the learned situation condition.

In the travel controller according to the present disclosure, theprocessor of the travel controller, at storing the object and thesituation, preferably further stores a rate at which the dangeravoidance action is detected in the case that the situation at detectionof the object during travel of the vehicle satisfies the situationcondition. The processor of the travel controller, at making the learnedsituation condition, preferably makes the learned situation condition sothat the situation condition corresponding to a higher rate of detectionof the danger avoidance action has priority over the situation conditioncorresponding to a lower rate of detection of the danger avoidanceaction.

A method for travel control according to the present disclosure includesdetecting one or more objects from an environmental image representingsurroundings of a vehicle capable of traveling under autonomous drivingcontrol satisfying a predetermined safety standard; detecting a lookingdirection of a driver of the vehicle from a face image representing aface region of the driver; and identifying an object in the lookingdirection of the driver out of the one or more objects. The methodfurther includes storing the identified object and a situation conditionindicating the situation at detection of the identified object in amemory in association with each other when a danger avoidance actionperformed by the driver to avoid danger is detected during travel of thevehicle; and changing the predetermined safety standard so that thedriver can feel safer, in the case where an object stored in the memoryis detected during travel of the vehicle under the autonomous drivingcontrol and where the situation at detection of the object satisfies thesituation condition.

The travel controller according to the present disclosure can controltravel of a vehicle so that its driver does not feel uneasy.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 schematically illustrates the configuration of a host vehicleequipped with a travel controller.

FIG. 2 schematically illustrates the hardware of the travel controller.

FIG. 3 is a functional block diagram of a processor included in thetravel controller.

FIG. 4 is a diagram for explaining identification of an object in alooking direction.

FIG. 5 illustrates an example of a situation table.

FIG. 6 is a flowchart of a first travel control process.

FIG. 7 is a flowchart of a second travel control process.

DESCRIPTION OF EMBODIMENTS

A travel controller that can control travel of a vehicle so that itsdriver does not feel uneasy will now be described in detail withreference to the attached drawings. The travel controller detects one ormore objects from an environmental image representing surroundings of avehicle capable of traveling under autonomous driving control satisfyinga predetermined safety standard regarding, for example, the vehiclespeed and the distance to an object in a surrounding area, and detects alooking direction of a driver of the vehicle from a face imagerepresenting a face region of the driver. The travel controlleridentifies an object in the looking direction of the driver out of theone or more objects. Thereafter, the travel controller stores theidentified object and a situation condition indicating the situation atdetection of the identified object in a memory in association with eachother when a danger avoidance action performed by the driver to avoiddanger is detected during travel of the vehicle. The travel controllerthen changes the predetermined safety standard so that the driver canfeel safer, in the case where an object stored in the memory is detectedduring travel of the vehicle under the autonomous driving control andwhere the situation at detection of the object satisfies the situationcondition.

FIG. 1 schematically illustrates the configuration of a host vehicleequipped with the travel controller.

The vehicle 1 includes an environmental camera 2, a driver monitoringcamera 3, and a travel controller 4. The environmental camera 2 and thedriver monitoring camera 3 are communicably connected to the travelcontroller 4 via an in-vehicle network conforming to a standard such asa controller area network.

The environmental camera 2 is an example of an environmental imagecapturing unit for generating an environmental image representing thesurroundings of the vehicle. The environmental camera 2 includes atwo-dimensional detector constructed from an array of optoelectronictransducers, such as CCD or C-MOS, having sensitivity to visible lightand a focusing optical system that forms an image of a target region onthe two-dimensional detector. The environmental camera 2 is disposed,for example, in a front and upper area in the interior of the vehicleand oriented forward, takes a picture of the surroundings of the vehicle1 through a windshield every predetermined capturing period (e.g., 1/30to 1/10 seconds), and outputs environmental images representing thesurroundings.

The driver monitoring camera 3 is an example of a driver image capturingunit for generating a face image representing a face region of thevehicle driver. The driver monitoring camera 3 includes atwo-dimensional detector constructed from an array of optoelectronictransducers, such as CCD or C-MOS, having sensitivity to infrared light,a focusing optical system that forms an image of a target region on thetwo-dimensional detector, and a light source that emits infrared light.The driver monitoring camera 3 is mounted, for example, in a front areain the interior of the vehicle and oriented toward the face of thedriver sitting on the driver's seat. The driver monitoring camera 3irradiates the driver with infrared light every predetermined capturingperiod (e.g., 1/30 to 1/10 seconds), and outputs images representing thedriver's face.

The travel controller 4 is an electronic control unit (ECU) including acommunication interface, a memory, and a processor. The travelcontroller 4 outputs control signals to a travel mechanism (not shown)of the vehicle 1, including an engine, brakes, and steering, so as tosatisfy a predetermined safety standard, and thereby executes autonomousdriving control of the vehicle 1. In addition, the travel controller 4detects an object in the driver's looking direction, based on anenvironmental image and a face image respectively received from theenvironmental camera 2 and the driver monitoring camera 3 via thecommunication interface, and stores the situation at detection of theobject. The travel controller 4 then changes the safety standard ofautonomous driving control of the vehicle 1, based on the situation atdetection of an object from an environmental image.

FIG. 2 schematically illustrates the hardware of the travel controller4. The travel controller 4 includes a communication interface 41, amemory 42, and a processor 43.

The communication interface 41 is an example of a communication unit,and includes a communication interface circuit for connecting the travelcontroller 4 to the in-vehicle network. The communication interface 41provides received data for the processor 43, and outputs data providedfrom the processor 43 to an external device.

The memory 42 is an example of a storage unit, and includes volatile andnonvolatile semiconductor memories. The memory 42 stores various typesof data used for processing by the processor 43, such as a set ofparameters for defining a neural network that functions as an objectclassifier for detecting an object from an environmental image, safetystandards used for autonomous driving control, and a situation table inwhich detected objects and the situations at detection of the respectiveobjects are associated with each other. The memory 42 also storesvarious application programs, such as a travel control program forexecuting a travel control process.

The processor 43, which is an example of a control unit, includes one ormore processors and a peripheral circuit thereof. The processor 43 mayfurther include another operating circuit, such as a logic-arithmeticunit, an arithmetic unit, or a graphics processing unit.

FIG. 3 is a functional block diagram of the processor 43 included in thetravel controller 4.

As its functional blocks, the processor 43 of the travel controller 4includes an object detection unit 431, a looking-direction detectionunit 432, an identification unit 433, a condition storing unit 434, acondition learning unit 435, and a standard changing unit 436. Theseunits included in the processor 43 are functional modules implemented bya program executed by the processor 43, or may be implemented in thetravel controller 4 as separate integrated circuits, microprocessors, orfirmware.

The object detection unit 431 detects an object in an area around thevehicle 1 by inputting an environmental image received from theenvironmental camera 2 via the communication interface into an objectclassifier that has been trained to detect an object.

The object classifier may be, for example, a convolutional neuralnetwork (CNN) including convolution layers connected in series from theinput toward the output. A CNN that has been trained using inputtedimages including objects as training data operates as an objectclassifier that detects an object from an image.

The looking-direction detection unit 432 detects the driver's lookingdirection from a face image received from the driver monitoring camera 3via the communication interface. The looking direction is expressed as ahorizontal angle between the travel direction of the vehicle 1 and thedirection in which the driver is looking.

The looking-direction detection unit 432 detects the positions of pupilsand corneal reflections in the driver's eyes included in the face imageby inputting the face image into a looking-direction classifier that hasbeen trained to detect the positions of pupils and corneal reflectionsof a light source. The looking-direction detection unit 432 then detectsthe looking direction, based on the positional relationship between thepupils and the corneal reflections.

The looking-direction classifier may be, for example, a convolutionalneural network (CNN) including convolution layers connected in seriesfrom the input toward the output. A CNN that has been trained usinginputted face images including pupils and corneal reflections astraining data operates as a looking-direction classifier that identifiesthe positions of pupils and corneal reflections.

The identification unit 433 identifies an object in the driver's lookingdirection out of the one or more objects detected from the environmentalimage.

FIG. 4 is a diagram for explaining identification of an object in alooking direction.

In the example of FIG. 4 , objects O1 (pedestrian), O2 (pedestrian), andO3 (another vehicle) and lane-dividing lines L1, L2, and L3 are detectedfrom an environmental image representing an area in front of the vehicle1. In addition, a looking direction DR1 is detected from a face image.The looking direction DR1 is a direction forming an angle α with respectto a travel direction DR0. The identification unit 433 identifies theobject O2 (pedestrian) at a position corresponding to the lookingdirection DR1 from the position of the vehicle 1 as an object in thedriver's looking direction.

The condition storing unit 434 stores a situation condition indicatingthe situation at detection of an object identified by the identificationunit 433 in the memory 42 in association with the identified object whenthe driver's danger avoidance action is detected during travel of thevehicle 1.

The danger avoidance action refers to an action performed on the vehicle1 by the driver to avoid danger. The danger avoidance action may be, forexample, an action performed by the driver during travel of the vehicle1 under manual driving control, such as pressing down the brake pedal todecelerate or stop, or rotating the steering wheel to avoid approachingan object. Alternatively, the danger avoidance action may be, forexample, an action performed by the driver during travel of the vehicle1 under autonomous driving control, such as pressing down the brakepedal or rotating the steering wheel to indicate the intention to avoiddanger, or pushing down a button so as to indicate such intention. Thecondition storing unit 434 detects a danger avoidance action byreceiving an operation signal via the communication interface 41 from anoperation unit connected to the in-vehicle network, such as the brakepedal, the steering wheel, or the button to indicate the intention toavoid danger.

The situation condition includes at least the distance from the vehicleto the object, e.g., the distance in the travel direction (lengthwisedistance) or the distance in the lateral direction (lateral distance)from the vehicle to the object. The situation condition may include theorientation of the object, road environment (e.g., the presence orabsence of a step between a sidewalk and a traffic lane, and the numberof lanes), the speed and the acceleration of the vehicle, and the timerequired for the vehicle to reach the object.

In the example of FIG. 4 , the condition storing unit 434 stores thelengthwise distance D1 and the lateral distance D2 to the identifiedobject O2 (pedestrian) and information indicating whether the directionDR2 in which the object O2 (pedestrian) is facing is the direction tothe host vehicle in the memory 42 in association with the object O2(pedestrian).

FIG. 5 illustrates an example of the situation table.

In the situation table 421, detected objects and the situations atdetection of the respective objects are stored in association with eachother. For example, at time T1, a pedestrian is detected in the driver'slooking direction; the driver's danger avoidance action is detected; andthe values indicating the lengthwise distance, the lateral distance, andwhether the object is facing the host vehicle, which are the situationcondition at this time, are stored. At time T2 different from time T1, apedestrian is detected in the driver's looking direction; the lengthwisedistance at this time satisfies the situation condition of thelengthwise distance at time T1; and the values indicating the lengthwisedistance, the lateral distance, and whether the object is facing thehost vehicle, which are the situation condition at this time, arestored. At time T3, a vehicle is detected in the driver's lookingdirection; and the values indicating the lengthwise distance, thelateral distance, the number of lanes of the road being traveled, andthe speed of the vehicle 1, which are the situation condition at thistime, are stored.

The situation table 421 only has to associate objects with situations,and the invention is not limited to data management in the form of atable such as illustrated in FIG. 5 .

The condition learning unit 435 makes a learned situation condition,based on a plurality of situation conditions stored at different timesin association with an object stored in the memory 42. The learnedsituation condition indicates a situation in which a danger avoidanceaction is detected at detection of the object.

To this end, the condition learning unit 435 preferably makes a learnedsituation condition so that a situation condition corresponding to ahigher rate of detection of a danger avoidance action has priority overa situation condition corresponding to a lower rate of detection of adanger avoidance action.

In the example of the situation table 421 illustrated in FIG. 5 , oftimes T1, T4, and T5 at which whether the pedestrian is facing the hostvehicle is YES, a danger avoidance action is detected at times T1 and T5and not detected at time T4 (the rate of detection is 67%). When whetherthe pedestrian is facing the host vehicle is NO (at time T2), no dangeravoidance action is detected (the rate of detection is 0%). Thus,regarding whether a pedestrian detected as an object is facing the hostvehicle, the condition learning unit 435 makes a learned situationcondition so that the situation condition, YES, corresponding to ahigher rate of detection of a danger avoidance action has priority overthe situation condition, NO, corresponding to a lower rate of detectionof a danger avoidance action.

In the example of the situation table 421, of times T1 and T2 at whichthe lengthwise distance to the pedestrian is 20 (m), a danger avoidanceaction is detected at time T1 and not detected at time T2 (the rate ofdetection is 50%). At time T4 at which the lengthwise distance to thepedestrian is 30 (m), no danger avoidance action is detected (the rateof detection is 0%). At time T5 at which the lengthwise distance to thepedestrian is 15 (m), a danger avoidance action is detected (the rate ofdetection is 100%). Thus, regarding the lengthwise distance to apedestrian, the condition learning unit 435 makes a learned situationcondition so that a situation condition (e.g., 15 (m)) corresponding toa higher rate of detection of a danger avoidance action has priorityover a situation condition (e.g., 30 (m)) corresponding to a lower rateof detection of a danger avoidance action. The learned situationcondition is made as a conditional expression such as “the lengthwisedistance to a pedestrian is not greater than 20 (m).”

The condition learning unit 435 may make a learned situation conditionby training a classifier that classifies the situations at detection ofthe objects stored in the situation table 421 according to whether thedriver's danger avoidance action is detected.

The classifier may be a support vector machine (SVM). The conditionlearning unit 435 inputs the objects and the situation conditions storedin the memory 42 in association with each other into a SVM to train it,thereby making a learned situation condition.

Alternatively, the classifier may be a neural network. The conditionlearning unit 435 inputs the objects and the situation conditions storedin the memory 42 in association with each other into a neural network totrain it, thereby making a learned situation condition.

The standard changing unit 436 changes the predetermined safety standardso that the driver can feel safer, in the case that an object stored inthe memory 42 is detected during travel of the vehicle 1 under theautonomous driving control and that the situation at detection of theobject satisfies the situation condition.

For example, in the case that a pedestrian is detected during travel ofthe vehicle 1 under the autonomous driving control and that thelengthwise distance to the pedestrian is 20 (m), the lateral distancethereto is 4 (m), and whether the pedestrian is facing the host vehicleis YES, the situation at detection of the object satisfies the situationcondition at time T1 at which a danger avoidance action is detected. Thestandard changing unit 436 then changes the predetermined safetystandard so that the driver can feel safer.

When the safety standard is the vehicle speed, the standard changingunit 436 lowers the vehicle speed in order that the driver can feelsafer. When the safety standard is the distance to an object in asurrounding area, the standard changing unit 436 lengthens the distancein order that the driver can feel safer.

When the situation at detection of an object satisfies a learnedsituation condition made by the condition learning unit 435, thestandard changing unit 436 determines that the situation at detection ofthe object satisfies the situation condition. For example, assume that alearned situation condition “the lengthwise distance to a pedestrian isnot greater than 20 (m), the lateral distance to the pedestrian is notgreater than 4 (m), and whether the pedestrian is facing the hostvehicle is YES” is made. When the lengthwise distance to a detectedpedestrian is 18 (m), the lateral distance to the pedestrian is 3.5 (m),and whether the pedestrian is facing the host vehicle is YES, thesituation at detection of the object satisfies the learned situationcondition; thus, the standard changing unit 436 may determine that thesituation at detection of the object satisfies the situation condition.

FIG. 6 is a flowchart of a first travel control process. The travelcontroller 4 repeatedly executes the first travel control process atpredetermined intervals (e.g., intervals of 1/10 seconds) during travelof the vehicle 1.

First, the object detection unit 431 of the travel controller 4 detectsone or more objects from an environmental image generated by theenvironmental camera 2 (step S11). The looking-direction detection unit432 of the travel controller 4 detects the looking direction of thedriver of the vehicle 1 from a face image generated by the drivermonitoring camera 3 (step S12).

The identification unit 433 of the travel controller 4 then identifiesan object in the driver's looking direction out of the detected one ormore objects (step S13).

The condition storing unit 434 of the travel controller 4 determineswhether the driver's danger avoidance action is detected (step S14).When a danger avoidance action is detected (Yes in step S14), thecondition storing unit 434 stores the identified object and a situationcondition indicating the situation at detection of the identified objectin the memory 42 in association with each other (step S15) andterminates the first travel control process.

When no danger avoidance action is detected (No in step S14), thecondition storing unit 434 terminates the first travel control process.

FIG. 7 is a flowchart of a second travel control process. The travelcontroller 4 repeatedly executes the second travel control process atpredetermined intervals (e.g., intervals of 1/10 seconds) during travelof the vehicle 1 under autonomous driving control.

First, the object detection unit 431 of the travel controller 4 detectsone or more objects from an environmental image generated by theenvironmental camera 2 (step S21).

The standard changing unit 436 of the travel controller 4 thendetermines whether the detected object is an object stored in thesituation table 421 in the memory 42 (step S22). When it is determinedthat the detected object is not an object stored in the situation table421 (No in step S22), the standard changing unit 436 terminates thesecond travel control process.

The standard changing unit 436 of the travel controller determineswhether the situation at detection of the object satisfies the situationcondition (step S23). When it is determined that the situation atdetection of the object does not satisfy the situation condition (No instep S23), the standard changing unit 436 terminates the second travelcontrol process.

When it is determined that the situation at detection of the objectsatisfies the situation condition (Yes in step S23), the standardchanging unit 436 changes the predetermined safety standard so that thedriver can feel safer (step S24) and terminates the second travelcontrol process.

By executing the first and second travel control processes in this way,the travel controller 4 can control travel of a vehicle so that itsdriver does not feel uneasy.

Note that those skilled in the art can make various changes,substitutions, and modifications without departing from the spirit andscope of the present disclosure.

What is claimed is:
 1. A travel controller comprising a processorconfigured to: detect one or more objects from an environmental imagerepresenting surroundings of a vehicle capable of traveling underautonomous driving control satisfying a predetermined safety standard;detect a looking direction of a driver of the vehicle from a face imagerepresenting a face region of the driver; identify an object in thelooking direction of the driver out of the one or more objects; storethe identified object and a situation condition indicating the situationat detection of the identified object in a memory in association witheach other when a danger avoidance action performed by the driver toavoid danger is detected during travel of the vehicle; and change thepredetermined safety standard so that the driver can feel safer, in thecase where an object stored in the memory is detected during travel ofthe vehicle under the autonomous driving control and where the situationat detection of the object satisfies the situation condition.
 2. Thetravel controller according to claim 1, wherein the situation conditionincludes at least the distance from the vehicle to the identifiedobject.
 3. The travel controller according to claim 1, wherein theprocessor is further configured to make a learned situation condition,based on a plurality of situation conditions stored at different timesin association with an object stored in the memory, the learnedsituation condition indicating a situation in which the danger avoidanceaction is detected at detection of the object, and, at changing thesafety standard, the processor determines that the situation atdetection of the object satisfies the situation condition, when thesituation at detection of the object satisfies the learned situationcondition.
 4. The travel controller according to claim 3, wherein atstoring the object and the situation condition, the processor furtherstores a rate at which the danger avoidance action is detected in thecase where the situation at detection of the object during travel of thevehicle satisfies the situation condition, and, at making the learnedsituation condition, the processor makes the learned situation conditionso that the situation condition corresponding to a higher rate ofdetection of the danger avoidance action has priority over the situationcondition corresponding to a lower rate of detection of the dangeravoidance action.
 5. A method for travel control, comprising: detectingone or more objects from an environmental image representingsurroundings of a vehicle capable of traveling under autonomous drivingcontrol satisfying a predetermined safety standard; detecting a lookingdirection of a driver of the vehicle from a face image representing aface region of the driver; identifying an object in the lookingdirection of the driver out of the one or more objects; storing theidentified object and a situation condition indicating the situation atdetection of the identified object in a memory in association with eachother when a danger avoidance action performed by the driver to avoiddanger is detected during travel of the vehicle; and changing thepredetermined safety standard so that the driver can feel safer, in thecase where an object stored in the memory is detected during travel ofthe vehicle under the autonomous driving control and where the situationat detection of the object satisfies the situation condition.